DOCUMENT RESUME 



ED 429 336 



EA 029 733 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 

REPORT NO 
PUB DATE 
NOTE 

CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Fuhrman, Susan H. 

The New Accountability. CPRE Policy Brief. 

Consortium for Policy Research in Education/ Philadelphia# 
PA. 

National Inst, on Educational Governance# Finance# 

Policymaking# and Management (ED/OERI) # Washington# DC. 

CPRE-RB-27 

1999-01-00 

14p . 

R308A60003 

CPRE Publications# 3440 Market Street# Suite 560# 
Philadelphia# PA 19104-3325 (single copy# free) ; Web site: 
http : //www . upenn . edu/gse/cpre/ ; e-mail : cpre@gse . upenn. edu 
Guides - Non-Classroom (055) 

MFOl/PCOl Plus Postage. 

^Academic Standards ; ^Accountability ; Educational 
Administration; Educational Change; Elementary Secondary 
Education; Performance; Quality Control; School 
Effectiveness 



ABSTRACT 



This policy brief reviews developments in standards-based 
reform and explores new ways for holding schools and districts accountable 
for discharging their missions. It draws on several Consortium for Policy 
Research in Education (CPRE) studies of accountability# many of which are 
still in progress. The report examines characteristics of new accountability 
systems and focuses on the following: performance# schools as the unit of 
improvement# continuous improvement strategies, inspections# more 
accountability categories# public reporting# and consequences attached to 
performance levels. It discusses issues in designing and implementing new 
accountability systems# asking such questions as "How is performance 
measured?" and "What is satisfactory progress?" The brief also describes ways 
in which to use fair measures in holding schools accountable# such as judging 
students' satisfactory progress rather than their absolute performance. It 
offers details on issues such as perverse incentives # complex indices of 
progress# what to do with students who fall in the middle of the 
distribution# states' capacity to remedy problems# stability and credibility 
of programs, public understanding, and student incentives. The report looks 
at ways in which new accountability systems work, and asserts that when such 
systems are well designed they can result in improved student achievement and 
more highly motivated teachers. (Contains 19 references.) (RJM) 
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The New Accountability 

by Susan H. Fuhrman 



A s part of standards-based reform, states and dis- 
tricts are designing new approaches to holding 
schools and districts accountable for discharging 
their missions. Virtually every state and thou- 
sands of districts are working on developing standards for 
student learning and aligning student assessments to those 
expectations. Most are taking the next step which is to use 
achievement of the standards as a basis for accountability. 

The new accountability approaches emerging from this work 
differ from more traditional systems with respect to one or 
more of seven factors' . District/school approval is being linked 
to student performance rather than compliance to regula- 
tions; accountability is focusing more on schools as the unit 
of improvement; continuous improvement strategies involv- 
ing school-level planning around specific performance tar- 
gets are being adopted; new approaches to classroom in- 
spection are being developed; more categories or levels of 
accreditation are being developed; school-level test Mores 
are being publicly reported; and more consequences are be- 
ing attached to performance levels. This policy brief reviews 
these developments and discusses issues arising from their 
design and implementation. It draws on several CPRE stud- 
iesofaccountability,mostofwhicharestill inprogress. There- 
fore a number of the findings we cite, particularly about the 
effects of emerging accountability systems, are tentative. 
However, since policymakers are actively designing and modi- 
fying accountability policies, we believe that conveying cur- 
rently available information about how they seem to be working, 
OQ even if that information is preliminary, is worthwhile. 



Characteristics of New 
Accountability Systems 

A Focus on Performance. States are trying to put more 
emphasis on student performance and less on compliance 
with regulations in their accreditation or certification of dis- 
tricts and schools. Traditionally, states have monitored com- 
pliance to input and process standards through self-reports 
and periodic visits. For example, state agencies have moni- 
tored pupil/teacher ratios through local forms specifying per- 
sonnel assignments and student enrollments, and crosswalked 
them with state certification data to monitor assignment of 
teachers by field. Minimal curricular requirements have been 
monitored through on-site inspection of syllabi and other pa- 
perwork, such as school board minutes, indicating adoption 
of specific curricula. 

More recently, the focus on student performance has led to 
the monitoring of outcome data such as test scores and gradu- 
ation rates. Outcome measures (either the level of perfor- 
mance or growth in performance) are now being used as 
criteria for accreditation. Not only are policymakers inter- 
ested in progress in the three R’s, but they are also including 
measures of performance in all core subjects in accountabil- 
ity systems. 

At the same time, policymakers in many states say they are 
trying to reduce regulations and compliance monitoring. The 
idea is to free schools fix»m the old compliance mentality and 
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to provide more flexibility so they can maximize student per- 
formance using new standards. 

Schools as the Unit of Improvement Whereas traditional 
accountability systems held districts responsible and focused 
on district-level activities such as employing teachers appro- 
priately and developing policies to implemait state directives, 
new systems focus on schools. Now measures of success 
are collected and reported school by school. Districts might 
also be held responsible for aggregate performance, as in 
Kentucky, but the important point is that data are now ana- 
lyzed and reported by school. In the new systems, the state 
might act — impose a consequence — on these school data, 
regardless of how good the district-level data look or how the 
district defines its own responsibilities vis a vis school per- 
formance. 

Continuous Improvement Strategies. Most of the new 
accountability approaches rest on state-determined perfor- 
mance standards or benchmarics of adequate progress. How- 
ever, one new approach involves school-site determination 
of and planning around specific performance targets, such as 
improved test scores in reading and math. Progress toward 
meeting locally set targets is the primary measure on which 
accreditation or state approval hinges. In Kansas, schools 
develop improvement plans with the assistance of a visiting 
team of peers within the first 1 8 months of a 5-year cycle. 
At the end of the cycle, the same (external) visiting team of 
3-4 peers returns to see whether the goals stated in the 
school’s plan have been achieved. 

Inspections. Some states are trying to focus their school 
and district visits on teaching and learning. Instead of paper 
reviews and central office visits, the new forms of inspection 
consist of lengthy peer visits that include classroom observa- 
tion and involve feedback and extensive discussions about 
practice. In addition to Kansas, just referred to above, Rhode 



Island is among the states using in-depth reviews. These 
new inspection approaches are sometimes likened to private 
agency accreditation, but the aims of the newer state ap- 
proaches are more ambitious with respect to providing for 
reflection on practice and attention to student work. 

More Accountability Categories. Categories of perfor- 
mance are being expanded to discriminate more discretely 
among districts and schools. Districts and schools no longer 
simply pass or fail . Rather, they earn ratings that vary along 
a continuum. For example, Mississippi has five levels. Tliose 
scoring high (levels 4 and 5) are judged to have high-quality 
programs offering the best opportunity to students. They are 
allowed some freedom fi-om regulatory policies like teacher 
certification, time and scheduling, and designing things like 
K-4 groupings. At the other end are level 1 districts, which 
are placed on probation. 

Public Reporting. More states are publicly reporting dis- 
trict and/or school test scores along with other outcome mea- 
sures such as attendance and dropout rates. According to 
the Council of Chief State School Officers (CCSSO), 47 states 
publicly report scores; 39 of them report school-level data. 
(CCSSO, 1998) 

Consequences Attached to Performance Levels. Increas- 
ingly, consequences are being attached to various accredita- 
tion levels and/or levels of performance. In addition to public 
reporting, which draws citizens into the process of evaluating 
progress and generates its own consequences for schools, 
some states also provide monetary rewards or other forms 
of tangible recognition, such as flags or pennants for schools 
meeting or exceeding performance targets. Currently 14 
states provide rewards, mostly monetary in nature (ECS, 
1997). 



About CPRE Policy Briefs 

CPRE Policy Briefs are published by the Consortium for Policy Research in Education (CPRE). The research reported 
in this issue was supported by a grant (No. OERI-R308A60003) to CPRE from the National Institute on Educational 
Governance, Finance, Policymaking and Management, Office of Educational Research and Improvement, U.S. Depart- 
ment of Education. The views expressed in this publication are those of its author and are not necessarily shared by 
the U.S. Department of Education, CPRE, or its institutional members. 



Sanctions now being proposed for poor performance reach 
beyond the punishments that states have previously employed 
for non-compliance. The ultimate punishment for non-com- 
pliance in the past has been loss of state aid, a step rarely 
taken. In the 1 980s, six states developed programs for state 
intervention in ^verely troubled school districts, sometimes 
referred to as “academically bankrupf ’ districts (OERI, 1 988). 
Additional states have adopted such approaches in the 1 990s; 
now 3 1 have sanction provisions in effect. Generally, a se- 
ries of interventions such as visits by technical assistance 
teams and the appointment of on-site monitors or conserva- 
tors precede the ultimate sanctions: state takeover, removal 
of local governance, or school closure or reconstitution with 
new faculty and perhaps reassigned students as well. For 
example, low performers in Oklahoma must develq) improve- 
ment plans and can ultimately face reassignment of faculty 
and financial penalty. In Maryland, sanctions can include 
reconstituting schools and bringing in new staff. 

Issues in Designing and 
Implementing New Accountability 
Systems 

Despite widespread interest in these new approaches to ac- 
countability, they are having a hard time getting off the ground. 
It appears that states and districts are encountering problems 
putting the new performance-based systems into effect Stud- 
ies of accountability systems in 1 0 states reveal a number of 
difficult issues that face policymakers and educators (Elmore, 
Abelmann and Fuhrman, 1 996; Massell, 1 998). Some of these 
concerns are technical^, some have to do with the capacity 
needed to implement new designs, and some are political. 

How is Performance Measured? Measuring student per- 
formance involves choosing a set of indicators and instru- 
ments. Most policymakers are mindful of the desirability 
of using multiple measures and not just relying on student 
test scores (e.g., see Oakes, 1 986). So, in addition to stu- 
dent achievement, they include indicators like graduation 
rates, post-secondary behavior, and attendance. 

Each of these indicators involves making decisions about 
the most appropriate data, but the most difficult and con- 
tentious choices have concerned student assessment instm- 
ments. For example, when a number of states began to 
design new accountability systems in the early 1990s, many 
policymakers hoped to include performance measures us- 
ing newer types of assessment formats. Performance tasks 



and open response formats were thought to provide better 
assessments of the knowledge and skills called for in new 
standards. However, questions about the reliability and 
validity of some innovative assessment instruments such as 
those used in Kentucky and Vermont led to hesitancy about 
building new accountability systems, especially those with 
significant rewards and sanctions, around them (Elmore, 
Abelmaim, and Fuhrman, 1996; Koretz, Deibert, and 
Stecher, 1 994; Catterall, Mehtens, Ryan, Flores, and Rubin, 
1 998; Kentucky Institute for Education Research, 1 995). 
Some early adopters of new forms put multiple choice for- 
mats back into their assessment systems, and by mid-de- 
cade, most were concerned about “balance,” about mixing 
some open-ended formats with more traditional item for- 
mats (Massell, Kirst, and Hoppe, 1 997). Limiting open- 
ended formats also reduces the cost and time requirements 
of assessments. This is important if states test each indi- 
vidual student, as many parents prefer, and not just samples 
of students. 

States also have to decide whether to develop their own 
assessments, designed specifically to address their stan- 
dards, or to purchase commercial assessments, many of 
which are advertised as being based on national curricular 
standards. These decisions have become more complex as 
commercial publishers have included open-response for- 
mats and performance tasks in their tests, seeking the bal- 
ance that states have tried to achieve in their own custom- 
designed assessments. 

Another important issue in measuring student performance 
is the determination of the standards of performance. 
Policymakers must not only pick a measure, but they must 
decide what is good or proficient performance on that mea- 
sure. It is important that these standards of performance 
not seem arbitrary to the public. When newer and less 
familiar assessments are used, it may not be easy to explain 
to the public why one level of performance is proficient and 
another is not. This is especially the case if more familiar 
measures (e.g., S ATs or ACTs) seem to provide divergent 
information, as they did in some Kentucky schools, for ex- 
ample. Avoiding a perception of arbitrariness requires not 
only clear explanations of the standards-setting process, but 
also provision of ocamples of items and performance at each 
level, so people can see for themselves how performance 
levels differ and howthey reflect different levels of knowl- 
edge and skills. 



Difficult Issues Facing 
Policymakers and Educators 

• How is Performance Measured? 

• What is Satisfactory Progress? 

• Holding Schools Accountable for V alue- Added 

• Perverse Incentives 

• Complexity and Trade-Offs with Fairness 

• What About the Middle of the Distribution? 

• State Capacity for Remedy 

• Stability and Credibility 

• Process Regulations Don’t Go Away 

• Public Understanding 

• Student Incentives 



What is Satisfactory Progress? lAany new state account- 
ability systems hold schools responsible for growth in student 
performance. But how much growth is reasonable to expect 
in each measurement period? Is it reasonable to expect that 
the same rate of growth can be sustained over a long period 
of time? And should the same amount of growth be expected 
for schools that start at different levels of performance? 

Early versions of these new systems used growth projec- 
tions developed through consultative processes, in the ab- 
sence of actual evidence about how much growth would be 
reasonable to expect in each measurement period. In Ken- 
tucky, for example, each school is expected to progress to 
the point where Ae average student is performing at a profi- 
ciency level on the state assessment in all core subjects at 
the end of a 20-year period; every two years a school must 
move a tenth of the way toward that. However, now that 
these systems and their assessment mechanisms have been 
in place for a few years, some policymakers are hying to set 
growth curves for schools based on empirical evidence of 
what is feasible. Policymakers should consult test publishers 
as well as other states and districts that have used the mea- 
sures they are considering in order to learn about the aver- 
age growth paths and variation in performance of schools 
starting at different levels. Drawing on these experiences, 
policymakers can determine what growdi e?q)ectations to build 
into their new accountability systems. 

Holding Schools Accountable for Value-Added. Many 
policymakers believe that it is unfair to hold schools account- 
able for factors beyond their control, such as variations in 
preparation associated with the socio-economic background 
of students. So they focus accountability systems on satis- 



factory progress, rather than on absolute performance — all 
the more reason for setting progress goals carefully. But 
focusing on growth, often called the “value-added” by the 
school, generates new controversies. For one thing, many 
account^ility systems compute growth by looking at changes 
between this year’s 4* grade scores and last year’s 4* grade 
scores. In other words, they measure achievement by look- 
ing at the difference in performance of two different student 
“cohorts.” But teachers know that classes vary in unpre- 
dictable ways. A better measure of “value-added” would 
track the performance of individual students’ scores over 
time from grade to grade, but that can be very expensive, 
requiring yearly testing of all students. If the student popula- 
tion is highly mobile, it is even more expensive. In addition, 
some people say that accepting modest increments of growth 
in schools starting out with a very low performance base 
might never get their students to the level where they can 
participate fairly in society. Rewarding schools for making 
such gains without motivating them to reach standards of 
proficiency could be setting lowered expectations for their 
students. Many states try to incorporate both growth and 
absolute measures in their systems for this reason. 

Measuring learning gains that can be attributed to individual 
schools is complicated immensely by student mobility. If stu- 
dent turnover is high, it is hard to hold schools accountable 
for growth as the students being tested change frequently. 
Schools showing low growth rates overall sometimes show 
much higher rates of growth for students who have been 
attending for two or three years. Policymakers can use sta- 
tistical adjustments to deal with this problem although it is 
unlikely that they can address it to everyone’s satisfaction. 

Perverse Incentives. Most educators know that they can 
bring school-level average scores up by working extra hard 
with students in a few areas of the assessment, or by focus- 
ing their efforts on those students poised to make the largest 
gains. To avoid such manipulations of the accountability sys- 
tem, which ostensibly is intended to improve the overall per- 
formance of all students, policymakers often try to build in 
safeguards. They monitor student exclusions to make sure 
that low-performers are not kept out of the testing pool and 
take security precautions so that no one can tailor test prepa- 
ration to a few items. In addition, they can require that the 
scores of the most low-achieving students must improve as 
well as the average scores of schools. Texas holds schools 
accountable for showing significant growth in the scores of 
sub-populations of students, so that educators will focus their 
efforts on each ethnic group. Similar procedures can be used 
to ensure that gender differences are not neglected. 
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Complexity and Trade-Offs with Fairness. In order to 
use multiple measures, focus on value-added, adjust for mo- 
bility and avoid perverse incentives, many states have devel- 
oped very complex indices to judge school performance and 
progress. Such formulas may be necessary to fairly account 
for all the factors influencing performance, but they are diffi- 
cult to explain to the public. Frequently policymakers them- 
selves do not understand them, and when various constituen- 
cies object to sanctions or complain about failure of a school 
to get a reward, they are unable to defend the system. 
Policymakers should hold themselves responsible for^^^at- 
ever accountability system they adopt and for making it com- 
prehensible to the larger public. They should balance com- 
plexity in the name of fairness with clarity in the name of 
legitimacy. 

What About the Middle of the Distribution ? Most of these 
emerging systems focus attention on the lowest-performing 
schools, which are subject to sanctions, and the highest-per- 
forming schools, which are eligible for rewards. They do 
little for those in the middle. Schools plcxlding along at a pace 
not sufficient to get a reward keep their noses out of trouble, 
but they are unlikely to get technical assistance (or at least 
none directly associated with the accountability system) that 
might boost them into reward categories. Schools that are 
not improving, but not declining, are likely to avoid sanctions 
in many systems; they are also likely to be ignored. Evi- 
dence of a lack of attention to the middle is the flip-flopping 
of schools between sanctioned status and non-sanctioned sta- 
tus. In some states, schools in trouble get enough attention to 
get them out of trouble, but then are left alone for a year or 
two without further help, and often slip right back into a sanc- 
tion-eligible category . 

State Capacity for Remedy. The reason schools in the 
middle of the distribution are likely to be ignored, and schools 
at the bottom frequently get insufficient help, is that states 
have limited capacity. Not only are agencies hard-pressed 
due to cuts in state funding, but they also often lack Ae kind 
of instructional expertise necessary to make a difference in 
troubled schools. To generalize, states employ a series of 
remedies for schools judged to be low-performing, ranging 
from technical assistance to takeover to actually closing the 
schools, by either redistributing students elsewhere or by re- 
opening it with new staff. But often the remedies are vague 
and the technical assistance is meager. One part-time per- 
son might serve as an advisor, for example. The new federal 
Comprehensive School Reform Design Program, which pro- 
vides $50,000 annually for schools employing external de- 



signs, might help low-performing Title I schools get assis- 
tance they wouldn’t otherwise. However, the most troubled 
schools may be too distressed to pick or profit fix)m a good 
design. They need an infusion of assistance nonetheless, 
and few states adequately budget for it It appears that many 
states fail to appreciate just how costly the remedy or sanc- 
tion side of the accountability system can be. They know 
how to cost out rewards, but tend to underestimate the cost 
of sanctions. 

Stability and Credibility. Political change is threatening to 
accountability systems that depend on educators’ faith that 
the consequences they are told to expect will really happen. 
Leadership turnover and change of heart — and the tendency 
to back away from severe sanctions at the end of the day, 
which has happened in some places — undermine the cred- 
ibility of these systems. An illustrative anecdote is the fa- 
mous “$300” problem in Kentucky. In the 1 980s, the legisla- 
ture passed and then failed to fund a promised bonus for 
teachers. The new accountability system enacted in 1990 
had to overcome this history and convince teachers that they 
really would get the rewards built into the new system. Skep- 
ticism was a major barrier to making the new system woiic. 
Legislators might consider putting reward dollars in escrow 
to reassure teachers that they really will be available in the 
future. 

Those Process Regulations Don ’t Go A w(^. Bearing down 
on performance makes sense if the state simultaneously re- 
laxes fiom process regulations that might restrict flexibility in 
reaching the new performance goals. Examples are man- 
dates to teach students or teachers specific topics, like Lyme 
Disease prevention, that might be worthwhile but that are not 
related to the state’s performance goals. Despite rhetoric 
about more autonomy in return for greater accountability, 
however, most states have not repealed much of the existing 
process regulation. Opposition from those attached to the 
status quo, comfort with traditional arrangements, and iner- 
tia, among other reasons, make it unlikely that policymakers 
will revisit old policy when they pass new policy. Some regu- 
lations protect powerful political constituencies, like special 
interest groups seeking a niche in the school curriculum. And 
many regulations are intended to ensure that minimal stan- 
dards are met in districts of varying wealth, location, size, 
and capacity. These regulations form the cornerstone of tra- 
ditional approaches to equity, and whether or not they are 
effective, it is still difficult to argue for their removal in the 
face of persistent differences in resources, teacher quality, 
and program offerings. 
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Public Understanding. Not only are new accountability 
systems likely to involve complex formulas, but also they of- 
ten entail many other unfamiliar elements: standards-based 
tests given to samples of students at key grades (as opposed 
to the familiar pattern of norm-referenced tests being given 
every year and yielding scores for each child relative to other 
children); new roles for states intervening in schools; and 
new publicity attached to performance. It is important that 
the public be educated about all of these aspects; without 
public support, new systems will wither over time. It is par- 
ticularly important that people accept the need for students 
to do well on the measures selected; it should be clear that 
performance on state assessments affects important life out- 
comes. Otherwise, parents will not encourage students to 
work hard in preparation, not understand why state money 
should go to rewarding schools that do well on such tests, 
and not support sanctions accorded to schools that don’t do 
well. 

Student Incentives. Most of the new systems do not em- 
bed consequences for students in their accountability ap- 
proaches. These systems are not designed to motivate stu- 
dents. Yet education is co-produced; teachers’ success is 
dependent on students’ efforts to learn. Many teachers ar- 
gue that ignoring student consequences is unfair. They can 
be penalized if the students don’t do well but there is nothing 
to motivate students to take the tests seriously. Gradually, 
some states are attaching promotion and graduation conse- 
quences to their new assessment measures, phasing in the 
requirements so that schools have a lot of time to give stu- 
dents adequate opportunity to learn the necessary material. 

International evidence suggests that curriculum-based exami- 
nations at the time of high school completion can spur higher 
overall achievement. Students fiom countries using medium- 
and high-stakes assessment systems outperform students fix)m 
other countries, at a comparable level of economic develop- 
ment, that lack such systems. In addition, qualifications for 
entry into secondary school teaching are also higher in na- 
tions with curriculum-based external exit examinations. Evi- 
dence from Canada shows this same pattern among that 
nation’s provinces. Not only did 1 3-year-olds from Canadian 
provinces with exit exams know more science and math- 
ematics than students in other provinces, they watched less 
TV and talked with their parents more about schoolwork. 
Schools in provinces with external exams were more likely 
to employ specialist teachers of mathematics and science; 
hire math and science teachers who had studied the subject 
in college; have high-quality science laboratories and experi- 
ments; schedule more hours of math and science instruction; 



and assign more homework in math, in science, and in other 
subjects. Clearly, Canadian external exams have not low- 
ered the quality of instruction, they appear to have enhanced 
it (Bishop, 1998). 

Even low-stakes curriculum-based exit exams such as the 
New York State Regents exams appear to have an effect. 
Holding student demographics constant. New Y ork State stu- 
dents do significantly better than students in other states On 
the SAT test and the NAEP math assessments (Bishop, 1 998). 
New York is currently phasing in a requirement that all stu- 
dents must pass the Regents exams to graduate, creating a 
high-stakes accountability environment for students. 

How Do New Systems Work? 

New accountability systems focus attention on performance, 
seeking to inspire school personnel to attend more to student 
achievement. What we don’t yet know is the quality and 
duration of the instructional changes that occur in responding 
schools. We don’t know whether such systems cause teach- 
ers and administrators to attend to the kind of capacity-build- 
ing that will enable them to improve instruction and learning 
overtime. 

Getting Attention. Evidence on effects comes from studies 
of high-stakes systems. Clotfelter and Ladd (1996) found 
higher pass rates in reading and math in Dallas schools, ^\^ere 
a complex reward system is in place, than in other Texas 
cities, even after adjusting for other reform efforts in those 
cities. In addition, the disaggregated, sophisticated informa- 
tion collected to operate the incentive program in Dallas pro- 
vided a usefiil managerial tool. 

CPRE has conducted interviews and large-scale surveys of 
teachers and principals in two research sites with new ac- 
countability systems: Nordi Carolina’s Charlotte-Mecklenburg 
School District and Kentucky (Kelley, Milanowski, and 
Heneman, 1998). We found that setting student achieve- 
ment goals for a school helps provide teachers with a focus 
for their work and increases the energy devoted to instruc- 
tion. New systems help channel teachers’ work to the most 
important goals of the system, largely those included in the 
performance measure, which, in the cases we are studying, 
is student achievement in the core academic subjects. 

Teachers placed value on the consequences included in these 
accountability systems, and those consequences helped mo- 
tivate them to work in more focused ways to produce irn- 
proved student achievement. In the Kentucky and Char- 
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lotte-Mecklenburg studies, we found diat teachers positively 
valued personal satisfaction from increasing student learn- 
ing, professional recognition for doing a goodjob, and receiv- 
ing a monetary bonus. Bonuses seemed to be less of an 
incentive for many teachers than a “thank you” for a job well 
done, but teachers tended to agree that the bonus was an 
important symbol of accountability or efficiency to the pub- 
lic. Negative outcomes, \^fiich were equally motivating, in- 
cluded increased pressure and stress to improve results, fear 
of being labeled as a “school in decline” and the accompany- 
ing professional embarrassment, loss offiieedom through some 
state directed assistance or “takeover,” and expanded woric 
hours. 

Additional findings about the bonus part of these programs 
were that teachers who reported being most motivated by 
the program and wished to see it continue were also those 
who were most dissatisfied with their current level of pay, 
those who felt it was fair to receive a bonus for improving 
student achievement, and those who felt that administration 
of the bonus process was fair. 

Our conclusion is that, contrary to the views offered by some 
commentators, money bonuses are valued by teachers and 
can be motivating, and sanctions such as school reconstitu- 
tion or identification as a school in decline are also valued 
(though negatively) and can also function to motivate. Both 
get teachers’ attention. 

Variable Responses. It is also clear from our work, how- 
ever, that teachers and schools vary in their responses to 
these accountability systems. Several factors help account 
for this variation. If schools had previously received a re- 
ward, the possibility of another positive outcome tended to be 
a primary motivator. If a school was in a non-improving 
mode, the negative outcomes seemed more salient. 

Much of the variation in response is related to individual and 
organizational capacities. Teachers believe that their knowl- 
edge and skill and school conditions are the critical elements 
that help them accomplish accountability goals. Principal lead- 
ership skills, the opportunity for feedback on results, curricu- 
lum alignment, professional development, and the sense of 
professional community at the school are more likely to be 
assessed positively in schools that accomplished their goals 
for improving student achievement than in schools that did 
not 

In fact, without strong leadership, a cohesive professional 
community, and access to good professional development, a 



school’s response may be short-term and superficial. On the 
whole, the schools operating under severe sanctions (recon- 
stitution and probation) that CPRE is studying in San Fran- 
cisco and Chicago do not appear to be making fundamental 
changes in their core processes. These schools seem to be 
placing considerable emphasis on test preparation. Some 
may incorporate some structural changes (such as breaking 
up into small schools) but few ^pear to be making extensive 
or deep efforts to rethink their instructional program (O’Day, 
in press). 

An important question for research is how these new state 
and local accountability systems interact with variations in 
school capacity. If, as previous research predicts and cur- 
rent studies confirm, factors such as teacher knowledge and 
skill, principal leadership, professional community, etc. “en- 
able” positive school responses to external accountability 
systems, how do we work to foster these conditions? In 
other words, how can accountability systems lead to effec- 
tive capacity-building? Are certain approaches to account- 
ability more likely than others to engender c^>acity-building? 
Over the short-term? Over the long-term? How else can 
policy provide for capacity-building in addition to establish- 
ing accountability systems? 

Relationship Between Internal and External Accountabil- 
ity Systems. One way to think about the relationship be- 
tween school capacity and new accountability systems is to 
ask about pre-existing conceptions of accountability within 
schools. What are the within-school or “internal” norms 
about accountability and how do they interact with the “ex- 
ternal” systems created by state or district policy? When 
new external systems, such as the ones we have been de- 
scribing, hit schools, can we expect variations in responses 
based on how schools already think about their responsibility 
for student learning? In other words, “When accountability 
knocks, will anyone answer?” In 20 public, parochial, inde- 
pendent and charter schools, CPRE found that internal ac- 
countability systems varied widely and were not necessarily 
related to the external accountability system (e.g., whether 
the school was a charter school). It is important to note that 
none of these schools were in Kentucky or other settings 
with strong new accountability systems (Abelmann and 
Elmore, widi Even, Kenyon, and Marshall, 1999).^ 

In the typical school in our sample, individual teachers’ con- 
ceptions of their responsibility exercised the greatest influ- 
ence over how schools addressed the accountability prob- 
lem. For the majority of schools we studied, questions about 
the collective purposes of the enterprise were answeredby 



1 the aggregation of individual teachers^ decisions. The views 

' of individual teachers about their capacity and that of their 

students determined \diat, if anything, was done to improve 
performance, not collective deliberation or explicit manage- 
ment decisions. In such circumstances, the school’s “solu- 
tion” to the accountability problem — to >^1iom, for ^\^lat, and 
how? — became simply a collection of individual, often idio- 
syncratic, judgments by teachers, growing out of their back- 
grounds, capacities, and individual theories about >^4iat stu- 
dents can do. These judgments were powerfully influenced 
by teachers’ preconceptions about the individual traits of their 
students, as well as the characteristics of students’ families 
and the communities from which they come, and typically 
uninformed by systematic knowledge of \diat these students 
might be capable of learning under different conditions of 
teaching. 

Hence, in these schools, teachers assigned the most power- 
ful causality in their own conceptions of responsibility to fac- 
tors over which they, as teachers, have little or no control, 
and they assigned the least powerful causality to those con- 
ceptions over which they have the greatest control, the con- 
ditions of teaching and learning in the school. Schools oper- 
ating in this mode were characterized by an emphasis on 
order and control (the one collective expectation on which it 
was possible to get consensus in an essentially isolated work 
environment) and by low expectations for student learning, 
usually wrapped in theories about the relative deprivation of 
students, their families, and their communities. 

In these schools, teachers and principals often dealt with the 
demands of formal external accountability structures (cur- 
riculum guidance, testing, and the like) either by incorporat- 
ing them in superficial ways — claiming, for ©cample, that they 
were consistent with existing practice when they clearly 
weren’t — or by rejecting them as “unrealistic” for the type 
of students they served. As noted, none of the schools in the 
20-school study were located in strong external accountabil- 
ity environments. Some were in districts or states in the 
early stages of developing stronger external accountability 
systems. The charter laws operating in these states were in 
the early stages of implementation, and these states hadn’t 
yet confronted the issue ofcharter renewal directly. So the 
observed weak effects of external accountability systems 
may simply be attributable to die specific policy environments. 

But, it seems highly unlikely that schools in which all ques- 
tions of accountability are essentially questions of individual 

i teacher responsibility will be capable of responding to strong, 

i 
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obtrusive external accountability systems in ways that lead 
to ^stematic, deliberate improvement of instruction and stu- 
dent learning. The idea that a school will “improve” its in- 
structional practice, and therefore the overall performance 
of its students, implies a capacity for collective deliberation 
and action that schools operating in die typical mode observed 
in this CPRE study simply did not display. Where virtually all 
decisions about accountability are decisions taken by indi- 
vidual teachers based on their individual conceptions of what 
they and their students can do, it seems unlikely that these 
decisions will mystically produce a strategy for overall im- 
provement for the school. So for schools operating in this 
mode, the key question for future research on the effects of 
external accountability systems is whether schools can, or 
will, respond by developing congruent internal expectations 
and accountability systems. And, perhaps more important, a 
related question is how these schools develop the capacity to 
generate these new internal norms and processes. 

Schools that have developed clear internal expectations and/ 
or accountability systems raise a different set of issues about 
the relationship between internal and external accountability. 
Our research suggests that such schools answer the ques- 
tion of ^\hat they are accountable for in very different ways — 
some focus on relatively low-level therapeutic purposes, some 
on higher-level aspirations for student performance. So for 
these schools, the issue is the degree of alignment between 
their internal expectations and accountability systems and the 
demands of external systems, as well as the level of conflict 
and accommodation that grows out of the confrontation be- 
tween internal and external accountability. Are schools that 
manifest some capacity to deal collectively with the internal 
accountability problem more likely to adapt and align their 
internal norms and systems to the requirements of external 
systems, or are they likely to be more resistant to changing 
their norms and systems? Do such schools have the neces- 
sary capacities to do the work of accommodating and adapt- 
ing new external requirements? The existence of collective 
internal expectations and accountability structures, in other 
words, does not necessarily predict exactly how a school will 
respond to new external requirements. These are questions 
to pursue in future research. 

New Accountability Systems: Getting Attention vs. Get- 
ting Capacity. The research that we have just summarized 
shows that new accountability systems can be motivating, 
especially in the presence of certain enabling conditions, and 
that internal, shared beliefs about the importance of student 
p)erformance are likely to be key enablers. 
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Figure 1 

A Model of Teacher Motivation 
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Previous research calls attention to both “will” (motivation) 
and “skill” (capacity) as influences on improving schooling 
(e.g., McLaughlin, 1987). In our studies of the influence of 
accountability systems, we have been working with a model 
of motivation developed from Expectancy Theory (Vroom, 
1 964) and Goal-Setting Theory (Locke and Latham, 1 990) 
(see Figure 1). The model suggests that a teacher will be 
motivated to try to reach the school’s student achievement 
goals to the extent that she (a) perceives a high probability 
that teacher effort will lead to students’ reaching achieve- 
ment goals (expectancy perception); (b) perceives a high 
probability that goal attainment will lead to certain conse- 
quences or outcomes such as a bonus award (instrumentality 
perception); and (c) places value, either positive or negative, 
on these outcomes. 

What we have learned is that new accountability systems 
seem to work primarily on (b) and (c); i.e., they relate goals 
(student achievement) to outcomes or consequences (both 
positive and negative) that teachers generally value. These 
are important contributions. As noted previously, teachers 
have often been skeptical about promises that reaching goals 
will have consequences; in the systems we have studied, this 
skepticism was overcome. Teachers believed that the bo- 
nuses or the sanctions would actually take place. And, they 
find both the positive consequences, including bonuses, and 
the negative consequences motivating. 

With respect to (a), however, the expectancy that teacher 
effort will actually lead to the desired level of student achieve- 
ment, accountability systems appear to provide only a/?ar/ 
of the answer. Goal-setting theory (Locke and Latham, 1990) 
suggests that the nature of the goals themselves can affect 
the effort-to-goal calculus. The clearer and more under- 
standable the goals, the more likely teachers will think that 

*7 




their effort will lead to reaching them. So, to the extent that 
new accountability systems include clear and specific goals, 
teacher expectancy may be affected. This finding certainly 
argues that attending to several of the design issues men- 
tioned previously — using meaningful measures, setting rea- 
sonable expectations of progress, and establishing compre- 
hensible and fair systems — ^are important. Further, when new 
systems include technical assistance, such as the help pro- 
vided by Distinguished Educators assigned to schools in de- 
cline or crisis in Kentucky, they are attending directly to fac- 
tors influencing expectancy; teacher knowledge and skills, 
curriculum alignment, and other enabling conditions. Reward 
schools in Kentucky that rebounded from decline status spe- 
cifically attributed their success to the assistance of a Distin- 
guished Educator (Kelley, 1998). 

Apart from the technical assistance provided to schools in 
trouble (v^iiich, as noted previously, is ficquently dropped when 
the school improves), accountability systems as currently 
designed do not directly work on the capacity issue — on the 
teacher competencies and school conditions that affect the 
effort-to-achievement calculus. Their design is based on one 
of two assumptions: either schools already possess the ca- 
pacity, but not the will to meet goals, or, once the goals are 
clear and valued consequences are attached, schools will shop 
for or find the capacity they need to meet the goals. 

However, we have already seen that the mere imposition of 
a new accountability system — even ones like the Kentucky 
and Charlotte-Mecklenburg systems where the goals are 
thought to be clear and outcomes both sure and motivating — 
does not unleash some hidden capacity. Schools clearly vary 
in their response to accountability systems, and that variation 
is strongly associated with existing school and teacher ca- 
pacities. As to whether accountability systems cause schools 
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to seek help and find or build the necessary capacity over 
time, only research over a longer period can tell. We are 
conducting such research, studying schools operating in es- 
tablished accountability systems as well as in systems just 
coming on line. We will look at different types of account- 
ability systems, including ones that attend to student incen- 
tives. Our work will examine both the demand side — whether 
schools seek capacity-building help — ^and the supply side — 
the adequacy and quality of capacity-building interventions 
available in the public, private, and non-profit sectors. If ac- 
countability systems do not create sufficient demand over 
time and/or if the existing supply of assistance is unsatisfac- 
tory (or unavailable to schools that lack sufficient fiscal re- 
sources), then states and districts will have to attend directly 
to capacity -building themselves if they want to increase sys- 
tem performance. They will not be able to rely on account- 
ability systems alone to leverage significant improvement in 
student achievement. 

Also clear from our research is that capacity-building efforts 
must attend to internal norms about accountability. Certainly 
teacher knowledge and skills are key, but so are teacher ex- 
pectations about student learning. Teacher beliefs about 
whether their students wil 1 benefit from their effort exercise 
a strong influence on the expectancy calculus. As we have 
seen fix)m our 20-school study of internal accountability, many 
schools’ teachers have no collective sense of responsibility 
for learning, and individual teachers are strongly influenced 
by what they think their students can or cannot do. They 
have had no experience with students from disadvantaged 
backgrounds learning challenging material; giving them such 
experiences and showing them how their children can learn 
to high standards are important aspects of capacity-building. 
In addition, the shared norms and processes underlying col- 
lective school improvement must be developed. 

Conclusion 

New accountability systems that are well-designed (with fair, 
comprehensible, meaningful, and stable features) are associ- 
ated with improved student achievement when adequate ca- 
pacity to improve instruction is present in schools or can be 
provided by an outside partner. Generally, teachers find the 
new systems motivating. But, in the absence of explicit at- 
tention to capacity, the new systems are insufficient ap- 
proaches to improving student achievement. Our continuing 
research will provide better information about the role of new 
accountability systems in encouraging sustained capacity- 
building and we look forward to reporting the findings in a 
future policy brief 
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End Notes 

1 . This discussion includes excerpts from Susan Fuhrman, 
“Evaluation of Performance in the United States: Changes in 
Accountability” prepared for the Organization of Economic 
Cooperation and Development, August 1 994. 

2. New developments in testing technology, such as internet- 
based computer-adapted testing, may alleviate some of the 
technical problems in the future (The Consortium on Renew- 
ing Education, November 1998. 20/20 Vision: A Strategy 
for Doubling America 's Academic Achievement by the Year 
2020. Nashville, TN: Peabody Center for Education Policy, 
Vanderbilt University). 

3. The remainder of this section draws heavily on language 
in Abelmann, et al., 1 999. 
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